Robot-assisted driving systems and methods

ABSTRACT

Systems and methods for driving a flexible medical instrument to a target in an anatomical space with robotic assistance are described herein. The flexible instrument may have a tracking sensor embedded therein. An associated robotic control system may be provided, which is configured to register the flexible instrument to an anatomical image using data from the tracking sensor and identify one or more movements suitable for navigating the instrument towards an identified target. In some embodiments, the robotic control system drives or assists in driving the flexible instrument to the target.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/201,787, filed Nov. 27, 2018, entitled “ROBOT-ASSISTED DRIVINGSYSTEMS AND METHODS, which is a continuation of U.S. patent applicationSer. No. 15/365,230, filed Nov. 30, 2016, entitled “ROBOT-ASSISTEDDRIVING SYSTEMS AND METHODS,” now U.S. Pat. No. 10,143,526, issued Dec.4, 2018, which claims the benefit under 35 U.S.C. § 119 to U.S.Provisional Application Nos. 62/261,301 filed Nov. 30, 2015, entitled“ROBOT-ASSISTED IMAGE-GUIDED NAVIGATION SYSTEMS AND METHODS,” and62/304,051, filed Mar. 4, 2016, entitled “ROBOT-ASSISTED IMAGE-GUIDEDNAVIGATION SYSTEMS AND METHODS.” The foregoing applications are herebyincorporated herein by reference in their entireties for all purposes.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference as if each individual publicationand patent application was specifically and individually indicated to beincorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates generally to flexible medical instruments, andmore particularly, to systems and methods for tracking and/orcontrolling the movement, location, position, orientation, or shape ofone or more parts of a flexible medical instrument disposed within ananatomical structure.

BACKGROUND

Minimally invasive procedures are increasingly being used by medicalpractitioners to diagnose and treat medical conditions. Compared to opensurgery, minimally invasive procedures involve smaller incision sizes,resulting in less injury to patients, improved recovery times, andreduced complications. A growing number of procedures are performedminimally-invasively using an access point (e.g., an incision)positioned remotely from the site of diagnosis or treatment. Forexample, increasingly, cardiovascular procedures such as aortic valverepairs and vascular stent implantations are performed by entering thepatient's vasculature via a small incision in the femoral artery.

Robotic surgical systems are well suited for minimally invasive medicalprocedures, because they provide a highly controllable yet minimallysized system to facilitate instrument navigation to areas that may liedeep within a patient. The Magellan® robotic catheter systemmanufactured by Hansen Medical Inc. (Mountain View, Calif.) is one suchrobotic surgical system; it includes a telescoping catheter systemformed of an inner elongate member and an outer elongate member. Boththe inner and outer members have multi-directional articulationcapabilities. Such a system is described, for example, in U.S. Pat. No.8,827,948. To navigate the robotic catheter using the system, thesystem's interface requires a user to direct the catheter's movement inmultiple degrees of freedom. The user must direct axial translation(i.e., insertion and/or retraction) as well as an articulation anglemagnitude (i.e., the bend) and articulation angle direction (i.e., theroll or roll plane) of both the inner and outer members. The user mustalso direct translation of a guidewire. While users can handleinstrument navigation relatively well when navigating in a constrainedspace such as a narrow blood vessel, it becomes much more challenging tonavigate in an organ, an ostium of a branch vessel, or other relativelyopen three-dimensional space. Navigation in such an open area forces theuser to understand the three-dimensional relationship of the instrumentrelative to the anatomical target and determine in which plane theinstrument will bend.

This task is difficult, in part, because navigating an instrumentthrough a lumen of the patient from a remote patient access point to thedesired site of a procedure requires manipulating the instrument withouta direct line of sight of the instrument. A tracking system may be usedto help locate the desired site of the procedure and visualize thenavigation of the instrument to the desired site of the procedure.Tracking systems allow the user to visualize a patient's internalanatomy and the location and/or orientation of the instrument within thepatient's anatomy.

Many visualization systems are not suitable for continuous real-timetracking of instruments though. For example, some systems such aspositron emission tomography (PET), X-ray computed tomography (CT), andmagnetic resonance imaging (MM) produce and combine many cross-sectionalimages of an object to generate a computer-processed image; such animage capture process is slow and movement within the photographed fieldduring the image capture process produces image artifacts that make suchsystems unsuitable for real-time tracking of moving instruments in abody. Additionally, some visualization systems such as X-ray CT andfluoroscopy emit potentially harmful ionizing radiation, and theduration of their use should be limited when possible. Direct endoscopicimaging (e.g., with an intraluminal camera) is suitable forpredominantly empty lumens such as the gastrointestinal tract but is notsuitable for blood-filled vasculature.

Tracking systems such as electromagnetic (EM) tracking systems and fiberoptic tracking systems provide a promising form of real-time instrumenttracking. EM sensing functions by placing an EM sensing coil (i.e., anEM sensor) in a fluctuating magnetic field. The fluctuating magneticfield induces a current in the coil based on the coil's position andorientation within the field. The coil's position and orientation canthus be determined by measuring the current in the coil. A single EMsensor is able to sense its position and orientation inthree-dimensional space with five degrees of freedom (i.e., in everydirection except roll). That is, the EM sensor is able to senseorientation in every direction except around the axial symmetric axis ofthe coil. Two EM sensors held fixed relative to each other on aninstrument may be used to sense all six degrees of freedom of theinstrument. In a navigation system employing EM tracking, an image of ananatomical space is acquired, the position and orientation of one ormore EM sensors on an instrument are detected, and the system uses aregistration between an EM sensor frame of reference and an anatomicalspace frame of reference to depict movement of the tracked instrumentwithin the imaged anatomical space. The use of EM sensors to trackmedical instruments and localize them to a reference image is described,for example, in U.S. Pat. Nos. 7,197,354 and 8,442,618. Fiber opticposition tracking or shape sensing devices are described, for example,in U.S. Pat. No. 7,772,541. In one example of fiber optic positiontracking, a multi-core optical fiber is provided within a medicalinstrument, with a light source coupled to one end of the optical fiberand a detector coupled to the opposing end. The detector is configuredto detect light signals that pass through the optical fiber, and anassociated controller is configured to determine the geometricconfiguration of at least a portion of the medical instrument based on aspectral analysis of the reflected portions of the light signals. Withsuch tracking systems, a medical practitioner can, in theory, observemovements of the instrument on a display and adjust user inputs asneeded to navigate the instrument to a target location.

In practice, users often struggle to navigate instruments to targetlocations with existing tracking systems. One cause of the problem isthat, for flexible instruments such as catheters, their shape inside theanatomy adjusts to the shape of the anatomy as the instrument isinserted. This shape does not always adjust uniformly or in a mannerthat is simple to predict, in part, because the stiffness of a flexibleinstrument is not uniform along the instrument. For example, in atelescoping catheter, a proximal segment of the outer member is stifferthan its articulation section, and the stiffness of the inner memberincreases if a guidewire is inserted inside. This lack of uniformity andpredictability can be problematic when inserting a flexible instrumentinto an anatomy, especially when using a tracking system with sensorsthat only track discrete point(s) on the instrument (such as EM trackingsensors). With such systems, it can be difficult to discern the entireshape of the instrument.

Users also struggle to navigate instruments to target locations becauserobotic catheter systems are not always intuitive to drive. Withflexible instruments that navigate through the anatomy, the instrument'stip position does not always follow the commanded position. This may bedue to distortion from contact with the anatomy or deformation of theinstrument due to articulation and insertion forces. This createsdifficulty in knowing the actual position of the instrument. Trackingand localization make knowledge of the instrument position in threedimensions more visible to the user, but many medical practitionersstill struggle to navigate the instrument to the desired anatomicaltarget even when a live-tracked instrument is displayed over an image ofthe anatomy.

The struggle is largely due to the nature of the two-dimensionalinformation being displayed to the practitioners. Some imaging systemshave incorporated 2-D/3-D image fusion systems, for example, asdescribed in U.S. Pat. No. 5,672,877. In one example, a fluoroscopicsystem can receive a pre-operative three-dimensional dataset from a CTor MRI and acquire two-dimensional images of the organ cavity or portionof the patient undergoing the interventional procedure. These systemscan then generate a 3-D/2-D fusion visualization of the organ cavity orportion of the patient based on the acquired two-dimensional image andthe three-dimensional image dataset. The three-dimensional image datasetis registered to the two-dimensional image. The three-dimensional imagedataset and the two-dimensional image are then displayed as a 3-D/2-Dfusion visualization, providing a 3-D model. However, even if athree-dimensional model is provided to help a user visualize theinstrument in space, the instrument representation is ultimatelyprojected onto a screen in two dimensions. Many users find it difficultto “think in three dimensions” (i.e., mentally convert two-dimensionalimages into the three-dimensional model).

Accordingly, there is a need for new and useful robotic systems thatcombine the capabilities of 3-D imaging and 3-D tracking whileaddressing the unique challenges of flexible instruments to assist usersin navigating instruments within the human body.

SUMMARY

Various aspects of the present disclosure address one or more of theneeds identified herein. For example, one aspect of the disclosure isdirected to a method for driving a flexible medical instrument in threedimensional space within an anatomy. The method, performed by a roboticmedical system, includes at least some of the following elements:acquiring one or more images pre-procedurally; acquiring one or moreintra-procedure images; registering the intra-procedure images with thepre-procedure images so that image frames from the pre-procedure imagesare matched to those from the intra-procedure images; inserting aflexible medical instrument intraluminally into a patient, theinstrument having one or more tracking sensors embedded therein;acquiring localization information for the instrument from the trackingsensors and tracking the location and position of at least a portion ofthe medical instrument using a tracking subsystem; superimposing thecurrent location and position of at least a portion of the medicalinstrument on the registered images; identifying a target in theregistered images; receiving a user command to drive the instrument;calculating a movement suitable to move the medical instrument from itscurrent location to or towards the target; and navigating or assisting auser in navigating the medical instrument to the target based, at leastin part, on the calculated movement.

In some embodiments, superimposing the current location and position ofat least a portion of the medical instrument on the registered imagesincludes overlaying a graphical representation of the instrument on atleast two different images to depict the instrument relative to theanatomy in different viewing angles. In some embodiments, the target isidentified within the registered images by the user. In someembodiments, the user provides the robotic medical system with anidentification of the target using a user input device. In someembodiments, the user command to drive the instrument is received fromthe user via a user input device. In some embodiments, calculating asuitable movement includes calculating one or a series of bends, rolls,insertions, and/or retractions needed to move the instrument from itscurrent location to or towards the target. In some embodiments,calculating the suitable movement includes determining a suitable changein instrument position in at least one degree of freedom. In someembodiments, receiving the user command includes receiving a usercommand to drive the instrument in a single two-dimensional plane, andcalculating the suitable movement includes identifying a suitablerotation of the instrument in a third dimension. In some embodiments,navigating the medical instrument to the target location includesnavigating a distal tip of a flexible inner member of the medicalinstrument to the target and driving a tubular outer member over theflexible inner member such that the tubular outer member generallyfollows over a path defined by the flexible inner member. In someembodiments, assisting the user in navigating the medical instrument tothe target includes controlling movement of the medical instrument in atleast one un-commanded degree of freedom while the user commandsmovement in one or more other degrees of freedom.

Another aspect of the present disclosure is directed to a method ofdriving a flexible instrument in three-dimensional space. The methodincludes: identifying a target location to which a user desires to drivea flexible instrument, the flexible instrument including a flexibleinner member and a tubular outer member; navigating a distal tip of theflexible inner member to the target; and advancing the tubular outermember over the flexible inner member such that the tubular outer memberfollows over a path defined by the flexible inner member. The navigationof the distal tip of the flexible inner or outer member to the targetmay be performed automatically or in conjunction with user inputs. Thenavigation of some embodiments includes determining and commanding oneor more instrument movements needed to navigate the flexible instrumentto or towards the target location. An additional aspect of thedisclosure is directed to a robotic medical system configured to performsuch a method. In some embodiments, the system includes a user inputdevice, an instrument driver, a controller in electrical communicationwith the user input device and the instrument driver, a trackingsubsystem, and a medical instrument comprising a guidewire, a flexibleinner member, and a tubular outer member. In some embodiments, thetracking subsystem includes position tracking sensors integrated intodistal portions of the flexible inner member, the flexible outer member,and the guidewire. In some embodiments, the system further includes adisplay screen.

Another aspect of the present disclosure is directed to a roboticmedical system for navigating a flexible instrument to a targetanatomical location. The flexible instrument has at least onecontrollable bending section and at least one position tracking sensorcoupled thereto. In various embodiments, the robotic medical systemincludes a user input device, an instrument driver, and a controller inelectrical communication with the user input device and the instrumentdriver. The controller of some embodiments includes a processor andmemory with instructions stored thereon, and the instructions, whenexecuted, cause the controller to: obtain a target location; receive oneor more user inputs from the user input device; and in response to auser input commanding a movement of the flexible instrument, determineand command movement of the flexible instrument in at least one degreeof freedom in order to help direct the instrument toward the targetlocation.

Another aspect of the disclosure is directed to a robotic medical systemthat includes a flexible instrument, a tracking subsystem, a userworkstation, an instrument driver, and a controller. In variousembodiments, the flexible instrument includes a proximal portion, adistal portion, and at least one controllable bending segment in thedistal portion. The tracking subsystem of various embodiments includesat least one position tracking sensor integrated at the distal portionof the flexible instrument. The user workstation of various embodimentsincludes a user input device and a display screen. The instrument driveris operably coupled to the flexible instrument and includes motors andother hardware configured to insert and retract the flexible instrumentand manipulate the at least one controllable bending segment. Thecontroller of various embodiments is in electrical communication withthe user workstation and the instrument driver. Moreover, the controllerincludes a processor and memory with instructions stored thereon,wherein the instructions, when executed, cause the controller to performa method that includes: signaling the display screen to display an imageof the flexible instrument over an image of an anatomical space,receiving an identification of a target location, determining one ormore instrument movements needed to navigate the flexible instrument toor towards the target location, and sending commands to the instrumentdriver to thereby control navigation of the flexible instrument in theanatomical space in accordance with one or more user inputs received viathe user input device and the one or more determined instrumentmovements. In some embodiments, the system further includes an imagingsubsystem.

An additional aspect of the disclosure is directed to a method forcontrolling navigation of a flexible instrument navigable in threedimensional space. The method is performed by a robotic medical systemthat includes a flexible instrument, an imaging subsystem, a trackingsubsystem having a tracking sensor integrated in the flexibleinstrument, an instrument driver, a workstation with a user input deviceand viewing screen, and a computer. In various embodiments, the methodincludes: acquiring an image with the imaging subsystem, acquiringlocalization information from the tracking sensor, registering thelocalization information to the image, overlaying the localizationinformation on the image for display to the user, receiving an inputfrom the user selecting a target on the image, and controllingnavigation of the flexible instrument toward the target. In someembodiments, controlling navigation of the flexible instrument includes:receiving user inputs directing movement of the flexible instrument in aplane, directing movement of the flexible instrument in that plane basedon the user inputs, and automatically determining and directing roll ofthe flexible instrument or movement in another plane.

A further aspect of the disclosure is directed to a method of providingnavigation assistance to a user who is navigating a flexible instrumentin an anatomical space of a patient. The method of various embodimentsis performed by a computerized system and includes: acquiring anddisplaying at least two images of the anatomical space, each imagedisplaying a different viewing angle; receiving a user designation of ananatomical target in each of the at least two images; calculating one ormore movements required for the flexible instrument to move from acurrent position to or toward the anatomical target; and utilizing thecalculated one or more movements to provide navigation assistance.

Additional exemplary configurations and advantages thereof will becomeapparent from the following detailed description and accompanyingfigures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a functional block diagram of one embodiment of arobot-assisted instrument driving system.

FIG. 2 illustrates a schematic representation of one embodiment of arobot-assisted instrument driving system provided within an operationroom setup.

FIG. 3 illustrates a schematic representation of one embodiment of arobot-assisted instrument driving system.

FIG. 4 illustrates a schematic block diagram of one embodiment of acontroller for a robot-assisted instrument driving system.

FIG. 5 illustrates a flow chart of one embodiment of a registrationtechnique of correlating a sensor reference frame to other selectivereference frames.

FIG. 6 illustrates a flow chart of one embodiment of a method ofrobotically assisting navigation of a medical instrument.

FIG. 7 illustrates a schematic representation of one embodiment of amedical instrument having tracking sensors integrated thereon.

FIGS. 8A-8E illustrate schematic representations of some embodiments oftracking sensors, as positioned within a medical instrument.

FIG. 9 illustrates a screen capture from the visual display of oneembodiment of a robot-assisted instrument driving system.

FIGS. 10A-10D each illustrates a screen capture from the visual displayof one embodiment of a robot-assisted instrument driving system. FIGS.10A-10C each provides two views as captured from a virtual biplanedisplay while FIG. 10D provides a magnified view of one view from thevirtual biplane display. Through the series of screen captures, FIGS.10A-10D together illustrate one embodiment of a method performed by arobot-assisted instrument driving system.

FIGS. 11A and 11B each illustrates a screen capture of one embodiment ofa virtual biplane formed from two views of a segmented cone beam CToutline overlaid on either a fluoroscopy image or a plain background.

FIGS. 12A and 12B illustrate one embodiment of a virtual instrument fromtwo different views: a side view and a front view, respectively. Bothviews may be provided within one embodiment of a virtual biplane.

FIG. 13 illustrates one embodiment of an anatomical image with a virtualinstrument superimposed thereon.

FIG. 14 illustrates a schematic representation of one embodiment of avirtual instrument along with various measurements of the virtualinstrument acquired by one embodiment of a robot-assisted instrumentdriving system.

FIGS. 15A-15E illustrate one embodiment of an anatomical image with avirtual instrument superimposed thereon. Through the series of figures,FIGS. 15A-15E together illustrate one embodiment of a method ofrobot-assisted instrument driving.

DETAILED DESCRIPTION

The foregoing is a summary, and thus, necessarily limited in detail. Theabove-mentioned aspects, as well as other aspects, features, andadvantages of the present technology will now be described in connectionwith various embodiments. The inclusion of the following embodiments isnot intended to limit the invention to these embodiments, but rather toenable any person skilled in the art to make and use this invention.Other embodiments may be utilized and modifications may be made withoutdeparting from the spirit or scope of the subject matter presentedherein. Aspects of the disclosure, as described and illustrated herein,can be arranged, combined, modified, and designed in a variety ofdifferent formulations, all of which are explicitly contemplated andform part of this disclosure.

Unless otherwise defined, each technical or scientific term used hereinhas the same meaning as commonly understood by one of ordinary skill inthe art to which this disclosure belongs.

As used in the description and claims, the singular form “a”, “an” and“the” include both singular and plural references unless the contextclearly dictates otherwise. For example, the term “an EM sensor” mayinclude, and is contemplated to include, a plurality of EM sensors. Attimes, the claims and disclosure may include terms such as “aplurality,” “one or more,” or “at least one;” however, the absence ofsuch terms is not intended to mean, and should not be interpreted tomean, that a plurality is not conceived.

The term “about” or “approximately,” when used before a numericaldesignation or range, indicates approximations which may vary by (+) or(−) 5%, 1% or 0.1%. All numerical ranges provided herein are inclusiveof the stated start and end numbers. The term “substantially” indicatesmostly (i.e., greater than 50%) or essentially all of a substance,feature, or element.

The terms “connected” and “coupled” are used herein to describe arelationship between two elements. The term “connected” indicates thatthe two elements are physically and directly joined to each other. Theterm “coupled” indicates that the two elements are physically linked,either directly or through one or more elements positioned therebetween.“Electrically coupled” or “communicatively coupled” indicates that twoelements are in wired or wireless communication with one another suchthat signals can be transmitted and received between the elements.

As used herein, the term “comprising” or “comprises” is intended to meanthat the device, system, or method includes the recited elements, andmay additionally include any other elements. “Consisting essentially of”shall mean that the device, system, or method includes the recitedelements and excludes other elements of essential significance to thecombination for the stated purpose. Thus, a device, system, or methodconsisting essentially of the elements as defined herein would notexclude other elements that do not materially affect the basic and novelcharacteristic(s) of the claimed invention. “Consisting of” shall meanthat the device, system, or method includes the recited elements andexcludes anything more than trivial or inconsequential elements.Embodiments defined by each of these transitional terms are within thescope of this disclosure.

Disclosed herein are robot-assisted, image-guided instrument drivingsystems and methods for navigating a medical instrument through ananatomical three-dimensional space where no direct line of sight isavailable to a medical practitioner. As shown in FIG. 1, in variousembodiments, the instrument driving system 10 includes an imagingsubsystem 16, a tracking subsystem 36, a user workstation 31, aninstrument driver 22, and a controller 34. Each of these elements andsubsystems is discussed in detail below. With these elements andsubsystems, the instrument driving system 10 is configured to assist innavigating a medical instrument 18 (not shown in FIG. 1) to a targetlocation in a three-dimensional anatomical space.

In various embodiments, the medical instrument 18 is a flexible and/orelongate medical device or any other tool that may be inserted into abody lumen. As non-limiting examples, the instrument may be a catheter,sheath, leader, probe, biopsy needle, aspiration tool, endoscope,optical fiber, guidewire, tool for delivering or implanting a stent orvalve, surgical tool, imaging tool, diagnostic tool, and/or therapeutictool. In various embodiments, the medical instrument is roboticallycontrolled. “Medical instrument,” “elongate instrument,” and “flexibleinstrument” are used interchangeably herein to refer generally to anyrobotically controlled instrument 18 configured for insertion into ananatomical lumen. In some embodiments, the medical instrument includes aflexible inner member and a tubular outer member. In some embodiments,the flexible inner member is a guidewire and the tubular outer member isa leader catheter. In other embodiments, the flexible inner member is aleader catheter and the tubular outer member is a sheath catheter. Instill other embodiments, the flexible inner member is a guidewire andthe tubular outer member is a sheath catheter. In some embodiments, aguidewire, leader catheter, and sheath catheter are provided.

In various embodiments, the elongate instruments 18 have one or morecontrollable bending sections or articulation sections. The bendingsections are manipulatable to change the direction of the tip of theflexible instruments as they are being advanced into the patient. Thedeflection or bending of the tip is sometimes referred to as the“articulation angle” and the corresponding tip direction is sometimesreferred to as the “heading direction”. The bending section may beconfigured to bend directly in multiple planes relative to itsnon-articulated state, or it may be configured to first bend in oneplane and be rotatable or rollable to reach another plane. Therotational orientation of the bending section is sometimes referred toas the “roll angle” or the “roll plane”. In various embodiments, theelongate instrument 18 has a proximal portion and a distal portion. Theterms “proximal” and “distal” are relational terms defined from theframe of reference of a clinician or robot arm. The proximal portion isconfigured to be positioned closer to the clinician or robot arm and thedistal portion is configured to be positioned closer to the patient oradvanced further into the patient.

In various embodiments, the anatomical space is a three-dimensionalportion of a patient's vasculature, tracheobronchial airways, urinarytract, gastrointestinal tract, or any organ or space accessed via suchlumens. Images of the anatomical space may be acquired using anysuitable imaging subsystem 16. Suitable imaging subsystems 16 include,for example, X-ray, fluoroscopy, CT, PET, PET-CT, CT angiography,Cone-Beam CT, 3DRA, single-photon emission computed tomography (SPECT),MRI, Optical Coherence Tomography (OCT), and ultrasound. One or both ofpre-procedural and intra-procedural images may be acquired. In someembodiments, the pre-procedural and/or intra-procedural images areacquired using a C-arm fluoroscope, such as described in U.S. Pat. No.8,929,631, the disclosure of which is herein incorporated by referencein its entirety. In the following discussion, the image and imageacquiring device (i.e., the imager) are often referred to using theterms “fluoroscopy image” and “C-arm,” respectively, but the inventionis not limited to use with fluoroscopy images; the same techniques applyto a variety of imaging subsystems.

In various embodiments, the tracking subsystem 36 tracks the medicalinstrument 18 as the medical instrument 18 progresses through theanatomical space. As used herein, a tracking subsystem 36 may also bereferred to as a position tracking system, a shape tracking system, or alocalization subsystem. The term “localization” is used in the art inreference to systems and methods for determining and/or monitoring theposition (i.e., location and/or orientation) of objects, such as medicalinstruments or tools in a reference coordinate system. Any suitabletracking system may be used. In many embodiments, the tracking subsystem36 includes one or more sensors placed on or in the medical instrument18 to enable tracking of the instrument 18. The tracking subsystem 36further includes a computerized tracking device configured to detect theone or more sensors and/or receive data from the one or more sensors. Insome embodiments provided herein, an electromagnetic (EM) sensing coilsystem is used. In other embodiments, a fiber optic tracking system orother tracking or localization system is used. The tracking sensor orlocalization sensor is often referred to herein as an EM sensor to avoidlisting numerous sensors for each embodiment, but it should beemphasized that any tracking or localization sensor, including a fiberoptic sensor, may be used.

A “sensed” medical instrument, as used at times herein, refers to aninstrument that has a position tracking sensor embedded therein and isbeing tracked. A “localized” medical instrument, as used at timesherein, refers to a sensed instrument that has been localized to areference coordinate system. As described in more detail further below,in some embodiments, the reference coordinate system may be an image ofthe patient or a part of the patient anatomy.

FIG. 2 provides one embodiment of an operating room setup that includesthe robotically-assisted instrument driving system 10. The depictedsystem 10 includes a table 12 upon which a patient 14 may be placed, afluoroscopy system or other imaging subsystem 16, and a catheter orother medical instrument 18. The depicted fluoroscopy system 16 includesa C-arm 28. A fluoroscopy panel 30 is mounted to the C-arm 28. The C-arm28 is selectively moveable during the procedure to permit various imagesof the patient to be taken by the fluoroscopy panel 30.

Attached to the table 12 is a robotic arm (also referred to as a setupjoint) 20 to which a robotic instrument driver 22 is coupled. One ormore splayers 24 may be mounted to the instrument driver 22. In someembodiments, the splayers 24 are coupled to or form a portion of themedical instrument 18. The medical instrument 18 of some embodimentsalso includes one or more pullwires disposed therein. The pullwires areattached to an articulation section of the medical instrument 18 andextend along a length of the instrument 18 to a proximal end. In suchembodiments, the splayers 24 are positioned at the proximal end of theinstrument 18. Each of the splayers 24 may include a pulley about whichone of the pullwires is wound and an interface for coupling with therobotic instrument driver 22. In some embodiments, the components areconfigured such that a motor in the robotic instrument driver 22rotationally drives an output shaft, which rotates the pulley of thesplayer 24 and thereby adjusts tension in the pullwire to articulate thearticulation section of the medical instrument 18.

The various components of the robotically-assisted instrument drivingsystem 10 are further visible in FIG. 3. One or both of the userworkstation 31 and the controller 34 may be remotely positioned (i.e.,free of a physical connection) with respect to the table 12. In someembodiments, one or both of the user workstation 31 and the controller34 are positioned in a separate room than the table 12. The userworkstation 31 includes a computer, a control console having a userinput device 33, and a visual display 35. The visual display 35 may be atouch screen, LCD screen, or any other suitable display configured topresent one or more images to a user. The user input device 33 mayinclude, but is not limited to, a multi-degree of freedom device havingmultiple joints and associated encoders. The user input device 33 mayadditionally or alternatively include a keyboard, joystick, buttons,switches, knobs, trackballs, touchscreen, or any other input devicessuitable for receiving commands from, and interfacing with, a user.

The controller 34 is a computing device. As shown in FIG. 4, thecontroller 34 includes electronics, including a processor 50, and memory52 having instructions stored thereon. The instructions, when executedby the processor 50, cause the processor 50 to perform various controlsmethods and execute various algorithms described elsewhere herein. Theprocessor 50 may be a general purpose microprocessor, a digital signalprocessor (DSP), a field programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), or other programmable logic device,or other discrete computer-executable components designed to perform thefunctions described herein. The processor 50 may also be formed of acombination of processing units, for example, a DSP and amicroprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suitableconfiguration.

The processor 50 is coupled, via one or more buses, to the memory 52 inorder for the processor 50 to read information from and writeinformation to the memory 52. The processor 50 may additionally oralternatively contain memory 52. The memory 52 can include, for example,processor cache. The memory 52 may be any suitable computer-readablemedium that stores computer-readable instructions for execution bycomputer-executable components. For example, the computer-readableinstructions may be stored on one or a combination of RAM, ROM, flashmemory, EEPROM, hard disk drive, solid state drive, or any othersuitable device. In various embodiments, the computer-readableinstructions include application software stored in a non-transitoryformat. The software, when executed by the processor 50, causes theprocessor 50 to perform one or more operations described elsewhereherein.

The controller 34 further includes one or more interfaces 54 (e.g.,communication databuses or network interfaces) for receiving user inputsfrom the user input device 33, transmitting images to the visual display35, and transmitting commands to the robotic instrument driver 22. Insome embodiments, the controller 34 is configured for bidirectionalcommunication with the robotic instrument driver 22, enabling thecontroller 34 to receive torque data or other feedback from theinstrument driver 22. In some embodiments, the controller 34 isphysically coupled to the user input device 33 and/or visual display 35of the user workstation 31. In some embodiments, the controller 34 isphysically coupled to the robotic instrument driver 22. In otherembodiments, the controller 34 is physically separate from, butcommunicatively coupled to the user workstation 31 and the roboticinstrument driver 22 via a wireless connection. A communication link 32transfers signals between the user workstation 31, the controller 34,and the robotic instrument driver 22. The communication link 32 may be awired or wireless communication link.

Each element of the robotic surgical system 10 positioned within theoperating suite may define a separate reference frame to whichlocalization sensors may be localized. More specifically, separatereference frames may be defined for each element of the robotic surgicalsystem 10. Such reference frames may include, for example, the followingshown in FIG. 2: a table reference frame TRF for the table 12, a setupjoint reference frame SJF for the setup joint or arm 20, a roboticinstrument driver reference frame RRF for the robotic instrument driver22, a splayer reference frame SRF for the splayer 24, and a fluoroscopyreference frame FF for the fluoroscopy panel 30. Additional referenceframes that may be defined in the system include: a patient referenceframe PRR for the patient 14, a reference frame FRF for a trackingsensor disposed in or on the elongate instrument 18, and a pre-operative3-D anatomical model reference frame AMF for the model depicted on thevisual display 35. In various embodiments, the robotic surgical system10 is designed to relate a coordinate system of the tracking sensor FRFof the elongate member 18 to either a fluoroscopy coordinate system FFor a pre-operative 3-D coordinate system AMF, as shown in FIG. 5. Therobotic surgical system 10 may employ a variety of registrationtechniques to register the FRF to the FF or AMF, such as those describedbelow or those described in U.S. Pat. No. 9,014,851 to Wong et al., thedisclosure of which is herein incorporated by reference in its entirety.

In various embodiments, the position or shape tracking sensorsincorporated into the medical instrument 18 allow for real-time sensingof the instrument's position (i.e., location, orientation, and/orshape). When the tracking sensor is integrated into the elongateinstrument 18 and localized or registered to the anatomy or an image ormodel of the anatomy such that the position of the elongate instrument18 is known relative to the anatomy, image, or model, apositionally-accurate representation of the instrument can be providedin the coordinate frame of the anatomical image or model. As theinstrument 18 moves through the patient, the tracking information of thesensor can be used to update the position of the elongate instrument 18relative to the anatomy, image, or model such that the representation ofthe elongate instrument can be displayed moving in real-time in ananatomical image or model. Additionally, with the instrument and theanatomical images provided in the same frame of reference, a targetanatomy may be identified in multiple fluoroscopy views to localize thetarget's position in three dimensional (3-D) space relative to theelongate instrument. An aspect of the disclosure provided herein is tomake use of this situation where the 3-D position of an instrument and atarget are known in real-time relative to a user's view of the patient'sanatomy in order to allow for novel navigation strategies not possiblewith traditional robotic or manual minimally-invasive instrumentnavigation. Robotic assisted driving, as provided herein, enhances thecapabilities of an instrument control or tracking system by allowing auser to easily navigate the instrument through the complex anatomy to atarget location without exposing the patient to excessive radiationduring the procedure.

One method of performing robotic-assisted navigation is provided in FIG.6. The method 600 is performed by a robotically-assisted instrumentdriving system, for example, the instrument driving system of FIGS. 1-4.In various embodiments, such a method 600 is performed by therobotically-assisted instrument driving system 10 in response toexecution of the instructions stored in the memory 52 of the controller34.

As shown in the depicted embodiment at S610, in some embodiments, theinstrument driving system acquires and displays two images of a relevantanatomy at different viewing angles. The images are acquired by theimaging subsystem, and any suitable imaging modality may be used.

As shown at S620, in some embodiments, the system localizes and displaysthe position of an elongate medical instrument relative to the images ofthe patient's anatomy. In some embodiments, this step includes acquiringlocalization information for the instrument from a tracking sensordisposed in or on the instrument, tracking the position of at least aportion of the medical instrument using a tracking subsystem,correlating the position of the instrument to the patient's anatomy, andsuperimposing a positionally-accurate representation of the instrumenton the two displayed images of the anatomy.

As shown at S630, in some embodiments, the system receives a userdesignation of an anatomical target in both images. The designation ofthe anatomical target in both images may be combined to compute a 3-Dposition (i.e., location, orientation, and/or shape) of an anatomicaltarget.

As shown at S640, in some embodiments, the system calculates onemovement or a series of movements required for the instrument to movefrom its current position toward, to, or through the target. Thiscalculation can be done continuously as the instrument moves through theanatomy or on-demand after a specific event or action is taken by theuser. In some embodiments, these calculations are computed for multipleinstrument components simultaneously (for example, for an inner memberand an outer member). In various embodiments, the calculated movementsinclude one or more of a magnitude and direction of articulation (e.g.,a bend and a roll). In some embodiments, the calculated movementsfurther include one or more of a magnitude and direction of axialtranslation of one or more instrument components (e.g., the innermember).

As shown at S650, in some embodiments, the system utilizes thecalculated movements to provide navigation assistance. The form ofnavigation assistance that is provided may vary widely betweenembodiments and/or modes. For example, the navigation assistance mayinclude: providing step-by-step navigation instructions to the user,controlling navigation in one or more degrees of freedom, rejectinguser-commanded movements that would navigate the instrument away fromthe target, driving the instrument towards the target while anauto-pilot indicator is actuated by the user, and/or driving theinstrument to the target in a fully-automated manner.

Each element of the assisted-driving method and the components that makeit possible are described in more detail below.

Tracking Sensors

In various embodiments, at least one tracking sensor is incorporatedinto the medical instrument to enable detection of the position (i.e.,location, orientation, and/or shape) of the medical instrument. In someembodiments, at least one tracking sensor is integrated into a flexibleinner member of a medical instrument; in some embodiments, at least onetracking sensor is additionally or alternatively integrated into atubular outer member of the medical instrument. For example, in theembodiment of FIG. 7, EM sensors are incorporated into the variouscomponents of the elongate instrument 700. While numbered uniquely, oneskilled in the art will appreciate that the medical instrument 18 ofFIG. 1 may be formed of any embodiment of an instrument described hereinand may include any of or all the features of the instrument 700 shownin FIG. 7. In the depicted embodiment, the medical instrument 700includes an outer sheath catheter 710, an inner leader catheter 720, anda guidewire 730. The sheath catheter 710 and leader catheter 720 eachhave a flexible distal portion, referred to herein as the articulationsection 716, 726, and a stiffer proximal portion, referred to herein asthe shaft 712, 722. Two five-degree of freedom (DOF) sensors 714 arelocated at the base of the sheath articulation section 716, two 5-DOFsensors 724 are located at the base of the leader articulation section726, and a single 5-DOF sensor 734 is located at or near the tip of theguidewire 730. The two EM sensors in each of the sheath and the leaderform a pair of sensor coils in each instrument. These pairs of 5-DOFsensors enable tracking of each of the leader 720 and the sheath 710 in6-DOF so that complete orientation, location, and heading are known. Inthe depicted embodiment of the guidewire 730, there is only enough spacefor a single 5-DOF sensor 734. In such embodiments, the guidewireposition and direction are sensed, but not the roll.

Combining two 5-DOF sensors into a single 6-DOF measurement (essentiallycalculating the roll angle of the instrument) can be accomplished in anumber of ways. In one embodiment, two 5-DOF coils 804 are combined intoa rigid assembly in a medical instrument 800 with known sensor locationsand with the two coils 804 configured to have different orientations oftheir symmetric axes 806, as shown, for example, in FIG. 8A. Thisprovides a strong or accurate 6-DOF measurement because the EM sensingtechnology is well-suited for sensing the heading, or symmetric axis, ofthe coils. There is, however, often inadequate space to place twononparallel coils into an elongate instrument such as a catheter. Insome embodiments, this limitation is overcome by spiraling the coils 814around a perimeter of the tubular elongate instrument 810, with a firstcoil spiral 814 a tilted slightly relative to a second coil spiral 814 bwithin the wall of the instrument 800, as shown, for example, in thecross-section of FIG. 8B. Such a configuration requires an elongateinstrument with a relatively thick sidewall. An alternative embodimentplaces two coils 824 nominally parallel in the elongate instrument 820to achieve the 6-DOF measurement, as shown in FIG. 8C. In someembodiments, the coils 824 are positioned diametrically opposite eachother across a cross-section of the elongate instrument 820, becausechanges in the relative position of the coils can be more accuratelydetermined with increased separation. In some embodiments, the coils arepositioned off-center (i.e., less than 180 degrees away from each other)due to the design of the elongate instrument. For example, in FIG. 8D,the placement of the central lumen 832 and pullwires 833 in the elongateinstrument 830 creates a non-uniformly thick sidewall 831, limitingplacement of the coils 834 to thicker portions of the sidewall. In someembodiments, such as in FIG. 8C, the coils 824 are parallel in boththeir orientation (e.g., axial alignment) and their position along thelength of the elongate instrument 820. With two parallel coils, combinedsensor measurement and specifically the roll direction can be calculatedby taking the difference in position between the two coil measurements.In some embodiments, the coils 844 may not be placed perfectly parallelalong the length of the elongate instrument due to manufacturingtolerances, as shown, for example, in FIG. 8E.

In some embodiments, the math to generate the 6-DOF measurement includesthe following, with reference made to FIG. 8E. First, a primary sensor844 a is used to find the point B′, which is axially in line with asecondary sensor 844 b and directly perpendicular to the orientation ofthe primary sensor 844 a. The vector from A to B′ defines the rolldirection of the sensor coordinate frame. A position of the “combinedsensor” can be computed as the midpoint of vector A→B′ if the sensorsare embedded in diametrically opposing locations of the instrument wall.If the sensors are not centered around the instrument shaft, as in FIG.8D, the relationship between the two sensors may be taken intoconsideration to adjust the position of the combined sensor. In someembodiments, the heading orientation of the combined sensor isdetermined either by taking the heading of the primary sensor (H_(A)) orby averaging H_(A) with the heading of the secondary sensor (H_(B)).

This method may cause a significant amount of roll error because the EMsensor measurements tend to have some error in their heading direction,and these errors are combined when producing the roll angle.Accordingly, to address this issue, in some embodiments, a low-passfilter is applied on the roll measurements. Elongate instruments in abody lumen generally do not roll very often or very quickly, so use of alow-pass filter does not significantly impact use of the elongateinstrument or sensors. The low-pass filter does stabilize any display ofsensor data that takes into consideration the roll information. In analternative embodiment, a hybrid method is used in which both the slightvariations in heading between the coils and the difference in positionof the coils is used to calculate the roll direction independently andthen combined.

In some catheter embodiments, for example, in FIG. 7, the EM sensorpairs provide position (x, y, z), heading (pitch and yaw), and rollorientation of the sheath and leader articulation sections. The EMsensor pairs may have any of the configurations described with regardsto FIGS. 8A-8E, or any other suitable configurations. Additionalelectromagnetic sensors may be added to different positions within themedical instrument to provide more information on the shape of theinstrument.

Registration

Registration is a process that requires relating the reference frame ofthe sensor FRF to another reference frame of interest. If the positionsof two or more objects are known in the same reference frame (i.e., aremapped to the same coordinate system), then the actual positions of eachobject relative to each other may be ascertained. Thus, with thisinformation, a user can drive or manipulate one of the objects relativeto the other objects. In various embodiments, the sensor reference frameFRF is registered to the fluoroscopy reference frame FF or to thefluoroscopic image or anatomical model AMF using a known registrationtechnique. There are many ways this registration can be performed. Insome embodiments, the sensor of the medical instrument is measured inrelation to the fluoroscopy system frame of reference FF. For example,in some embodiments, a sensing probe is used, which has an EM sensor anda radiopaque marker located in the same physical location on the probe.The sensing probe is placed into the field of view. The 2-D position ofthe probe is designated by the user in the fluoroscopy field of view(FOV) in images obtained at two different C-arm roll angles. Theposition of the probe may be designated by the user in three or moredifferent locations. These measurements are then used to sync the sensorlocation measurements with the selected fluoroscopy locations. In thisway, the EM coordinate system is registered to the fluoroscopycoordinate system FF. In most interventional procedures, the referenceframe of interest is the visualization frame. The visualization frame isthe frame that the user (e.g., a physician) is viewing, and it mayinclude a patient or a live 2-D or 3-D image. The live image may beacquired via fluoroscopy, ultrasound, digital subtraction angiography,live fluoroscopy with a contrast injection into the bloodstream, orother live imaging source. Using such techniques, the goal ofregistration is to determine the relationship of the frame of referenceof the sensor FRF relative to the frame of reference of the patient PRFor to a 2-D or 3-D image or model of the patient.

Virtual Instrument

Various methods of tracking and registering the elongate flexibleinstrument are described above. In various embodiments, once ananatomical image is acquired, the sensor on the elongate instrument istracked, and the sensor's frame of reference is registered to theanatomical image's frame of reference, a representation of the elongateinstrument is displayed on the anatomical image to facilitate a user'sability to visually track progress of the elongate instrument in theanatomy. The process for displaying the elongate instrument on the imagewill now be described.

The tracked instrument is simulated by rendering it with 3-D computergraphics and displaying, overlaying, or superimposing it on storedfluoroscopy images. One example of a simulated rendering of an elongateinstrument 920 superimposed over the anatomy 910 captured in a storedfluoroscopy image 900 is provided in FIG. 9. This simulated elongateinstrument 920 is known as the virtual instrument or the virtualcatheter. In some embodiments, the location, orientation, and shape ofthe virtual instrument 920 are estimated based on commanded data. Insome embodiments, sensor measurements are used to improve the quality ofthe simulation and generate more accurate instrument shapes that areusable in clinical settings. In such embodiments, one or more of thelocation, orientation, and shape of the virtual instrument 920 isdetermined with the aid of tracking sensors. The current locations andorientations of the tracking sensors in an anatomical space are knownfrom received sensor measurements. The fixed location and orientation ofthe sensors in each elongate instrument are also known. From these knowndata points, a virtual instrument 920 can be drawn that passes throughthese points. The total lengths and insertion distances of the variouscomponents of the elongate instrument are also known. Robotic movementsof each component are tracked and this movement can be used toextrapolate the instrument shape between the sensor positions. Therotational orientation of the elongate instrument may also be determinedfrom the sensors as described above to provide an entire 3-Dreconstruction of the elongate instrument. One method for displaying thevirtual instrument involves using spline curves to interpolate the shapeof the elongate instrument between sensors. This method is purelygeometric and therefore does not capture the characteristic behavior ofa real instrument. Another method involves using a physics-basedsimulation to model an elongate instrument. In one embodiment, aninstrument model comprises a series of points connected such that theymaintain realistic positions relative to one another. The virtualinstrument seen by the user is rendered as a 3-D object that follows apath through the series of points.

In some embodiments, this virtual instrument information may bedisplayed to the user to help the user navigate. For example,instinctiveness indicators 930 such as the ring with colored cones shownin FIG. 9 may be added to the virtual instrument 920 and used to signalto the user which direction the instrument will bend when a specificuser input command is activated. In the provided embodiment, thedirectional cones are on opposing sides of the ring (i.e., 180 degreesapart). Directional indicators of any distinguishing colors or shapesmay be used. In some such embodiments, corresponding directionalindicators may be placed on the user input device. The virtualdirectional indicators are continuously updated with the position of theelongate instrument to represent the direction the instrument would bendif the corresponding directional indicator on the user input device isactivated. For example, in one embodiment, a ring is provided around thevirtual instrument 920 with an orange cone and an opposing purple cone.On the user input device, a left button or left side of the joystick maybe marked with the orange mark. Activation in this direction would bendthe elongate instrument in the direction of the orange indicator onscreen. On the user input device, a right button or right side of thejoystick may be marked with the purple mark, and activation in thisdirection would bend the elongate instrument 180° from the firstdirection. Such an embodiment provides for more instinctive driving thansimple “right” and “left” activation buttons, because the elongateinstrument may rotate as it is advanced through the anatomy and theviewing angle of the C-arm may also rotate so it cannot be assured thatbending the instrument to the left with the input device would result inthe instrument bending to the left in the viewing plane. The presence ofthe 6-DOF position tracking sensors in the tip of the instrument may beused to communicate the actual roll orientation of the instrument tip inthe given viewing plane. In some embodiments of robotic assisted driving(described in more detail below), the colored cones on the ring (orother instinctiveness indicators) are augmented to include another shapeor other indicator to indicate which direction the system recommendsthat the instrument be articulated to aim towards the target.

In simulating telescoping catheters using a physics model, it is oftenadvantageous to treat multiple catheters as a single elongated object asit requires less computation. However, this model does not accuratelycapture the interaction between the catheters, introducing unrealisticconstraints in the simulation. For example, the model would be subjectto a large amount of torque if it were to match sensor measurementsexactly, because real catheters have room to slightly roll relative toeach other. This often leads to instability in simulation. In order toresolve the issue, in some embodiments, the model may use only a subsetof sensor measurements to reduce the risk of over constraining themodel. For example, the roll measurement at the instrument tip may notbe rigidly enforced.

In another embodiment, the accuracy of the virtual instrument 920 may beimproved by tracking the elongate instrument via computer vision in afluoroscopy image. Computer vision techniques to track catheters havebeen described, for example, in US Publ. No. US2016/0228032, thedisclosure of which is herein incorporated by reference. The similarityof the fluoroscopic instrument and the virtual instrument can be used togenerate bias forces to move the physics model closer to the realinstrument shape. In another embodiment, fiber optic shape sensingsensors may be used to estimate the shape of the virtual instrument. Ina further embodiment, the commanded robotic instrument insertion lengthor the commanded angle and heading orientation of an instrument may betracked and compared to measured instrument position and heading basedon sensor data and the delta may be used to update the physics modelaccordingly.

In some embodiments, a 3-D model of the anatomy is generated frompre-operative imaging, such as from a pre-op CT scan, and the instrumentmodel interacts with the anatomy model to simulate instrument shapeduring a procedure. For example, in one embodiment, the intersection ofthe instrument shape with the geometric model of the anatomy producesforces that are included in the simulation of the instrument. Inaddition, the time history of sensor locations provides insight as tothe shape of the anatomy or the possible shape of the instrument. As aninstrument passes through blood vessels, the instrument will oftenstraighten or deform the anatomical shape. By tracking the path of theinstrument through the anatomy over time, the relative shape of thedeformed vessels may be determined and both the instrument model and theanatomical model may be updated.

Interpolated instrument shapes become less accurate in parts of theinstrument far away from sensors. When the virtual instrument shapedeviates from the actual shape, physicians may inadvertently act on theincorrect information. Therefore, in some embodiments presented herein,a measure of confidence is displayed for each section of the virtualinstrument shape so that physicians can make informed decisions. Thismeasurement of confidence is guided by a few principles: the closer to asensor, the higher the confidence; the higher the curvature betweensensors, the lower the confidence; the greater the difference betweenthe known and measured sensor-to-sensor distance, the lower theconfidence; and the greater the difference in sensor orientations, thelower the confidence.

In some embodiments, the confidence measure in part of the virtualinstrument is shown non-numerically. For example, in one embodiment, thedegree of transparency in the virtual instrument 920 corresponds to themeasure of confidence. Part of the instrument may be made fullytransparent if the confidence is sufficiently low so that thequestionable portion of the instrument is hidden from the user.Alternatively, low-confidence may be represented by changing the coloror texture of a part of the instrument or by adding animation, such asflashing or scrolling texture. In another embodiment, a flashing icon,such as a radiation icon, may be displayed beside the fluoroscopy imageto urge the use of fluoroscopy when the confidence falls below athreshold value. It may start flashing when confidence drops as a way ofsuggesting that the user use fluoroscopy to acquire an updated image ofthe anatomy and the instrument. Low-confidence portions of the virtualinstrument may flash in time with the radiation icon to better associatelow confidence with the need for fluoroscopy.

Virtual Biplane

In various embodiments provided herein, a visualization mode called a“virtual biplane” is provided. In a virtual biplane, the virtualinstrument is overlaid on the standard primary image and also on asecondary reference view. The concept of a virtual biplane is introducedin US Publ. No. 2015/0223902 to Walker et al., the disclosure of whichis herein incorporated by reference in its entirety. Displaying arepresentation of the instrument updated in real-time, overlaid on twodifferent views of the anatomy is analogous to what a user would see ina biplane fluoroscopy system. However, as contemplated herein, thebiplane view is not an actual live biplane view, but rather, asimulation of the sensed instrument superimposed on the anatomicalimages. Therefore, it is known as a virtual biplane mode. The catheteror instrument that is displayed, overlaid, or superimposed on theanatomical image is referred to as the “virtual instrument” or “virtualcatheter” as described above. In the virtual biplane, the virtualinstrument is depicted in two different views of the anatomicalbackground. In some embodiments, both provided views utilizefluoroscopy. In some embodiments, the virtual biplane includes a firstfluoroscopic view with an image of the sensed medical instrumentoverlaid or superimposed on top of the fluoroscopic view. This may be alive fluoroscopic view or a previously acquired fluoroscopic view. Thecommercially available fluoroscopic systems have the capability ofacquiring and storing images. These images may then be displayed asreference images at any point during the procedure. The virtual biplaneembodiment presented here also includes a second view, which may be areference view, for example, a previously-acquired view obtained viafluoroscopy at a different angulation of the C-arm.

In one embodiment, the first and second view may be shown at differentmagnifications. For example, the first view may show an image at a lowermagnification so that more of the instrument and anatomy is seen to helpthe user understand the global position in the patient whereas thesecond view may be a zoomed in or magnified view of an area of interest,usually in a different projection from the first view.

As the medical instrument is moved or manipulated through the patient,the tracking sensor in the instrument tracks its movement and thevirtual instrument is updated in both views. This provides live 3Dtracking of the instrument displayed against images of the anatomy. Theposition sensor information is registered to each image so that as theimage changes (for example, due to a movement in the C-arm), the systemcan calculate where the sensor measurements line up with the updatedimage. At any point during the procedure, the user may change theanatomical images used for the virtual biplane. For example, if aphysician is attempting to target a first vessel pointing directlyanteriorly (i.e., toward the front of the patient), a lateralfluoroscopic projection might be preferred for at least one of the viewsso that the vessel is perpendicular to the viewing plane, whereas if asecond vessel is pointing partly anterior but partly to the side of thepatient, than the physician may wish to change over to a more obliquefluoroscopic projection.

A problem with said overlays is that the reference image shows thevessel anatomy at a specific instant in time. If the physicianintroduces a very inflexible or rigid instrument, the anatomy isdeformed, and if the patient moves on the table, the overlay is nolonger aligned. If said deformation or misalignment is not corrected inthe overlaid reference image, an imprecision or a discrepancy ariseswhen the reference image is superimposed. This can lead to uncertaintiesin navigation during an intervention in which the overlay serves as anavigation aid. Therefore, in various embodiments, the physician isprovided with the option of refreshing the image by taking another liveimage at any point during the procedure.

In some embodiments, the images are acquired using a C-arm fluoroscope,with one viewing angle acquired prior to the procedure and the otherviewing angle acquired intra-procedurally. In some embodiments, bothviews are displayed simultaneously, for example, adjacent to each other.In other embodiments, one or more of the images are generated by animaging system that overlays a registered pre-operative orintra-operative 3-D image (e.g., 3-D rotational angiography or cone beamCT) on a live image. In still other embodiments, a pre-operative orintra-operative 3-D image is acquired and displayed, which is notregistered or overlaid onto live imaging.

An example of a virtual biplane is provided in FIGS. 10A-10C. In someembodiments, two views are displayed on a split screen. In otherembodiments, two display screens are provided, each displaying adifferent view. In the example images captured in FIGS. 10A-10C, theleft view directly corresponds to the C-arm angle. If the C-arm isrotated, then this view, including the depicted EM sensing informationwill change; as the C-arm moves, the EM indicators and virtualinstrument will update according to that new C-arm angle. In someembodiments, if the user steps on the fluoro pedal of a fluoroscopicimaging subsystem, the image directly related to the C-arm angle willupdate based on the live fluoro image. In some such embodiments, if theuser releases the fluoro pedal, the last image will be used as areference image when driving an EM sensed instrument. In suchembodiments, the fluoro image may get out of sync with the EM sensordata if the C-arm is rotated but the user is not stepping on fluoro. Insuch embodiments, a visual indicator may be presented to the user toindicate that the fluoro image is no longer relevant. For example, insome embodiments, the outdated fluoro image may be blackened or given ahue or color or icon to show that it is old information. In otherembodiments, a sequence of fluoro images at different angles can beacquired and stored, possibly using a predefined C-arm motion to acquirethem, and the reference image can be updated based on C-arm motion evenwhen the user is not stepping on fluoro.

In some embodiments of the virtual biplane, the second image is always astored image associated with a particular angulation of the C-arm. Theparticular C-arm angulation is provided and used to allow the sensedinstrument information to update live according to that stored view. Inthe embodiment of FIGS. 10A-10C, the right image is the stored referenceimage. The stored reference image is a fixed snapshot of a fluoro imagetaken sometime in the past, which corresponds to the particular C-armangle. The user can use this stored background image as a reference orroadmap as they are driving. Different visual indicators may be used toshow that this is a stored image, such as colors, hues, or icons. At anytime, a button or other user input device can be selected to store thelive image and C-arm angulation for use as the stored reference in thefuture.

In some embodiments, it is possible to store multiple reference imageseach associated with a different respective C-arm angulation. A user orthe system may be able to select between the multiple reference imagesand use different ones at different times without using additionalradiation. For example, in some embodiments, the stored image consistsof a sequence of images at different C-arm angles. In such embodiments,the sequence of images may be sequentially displayed as the C-arm moveseven if the imaging is not live. Alternatively, in such embodiments, thedisplayed image selected from the sequence of images at different C-armangles may be chosen by the user through a user interface that allowsthe user to modify the viewing angle of the second image. In otherembodiments, the second image may automatically change to display imagesfrom various C-arm angles in a cyclic or periodic fashion providing ananimation of the live EM information that provides the user with morethree-dimensional information about the shape of the medical instrumentor anatomy.

In various embodiments, the images stored in the virtual biplane areoften views directly from the fluoroscopy system or other imagingsystem. In some embodiments, it may also be possible to include overlaysfrom the fluoroscopy system or other imaging system. Such a feature maybe helpful in certain workflows, for example, when the user wants to doa contrast injection and store the image during a contrast injection ordigital subtraction angiography (DSA) to show the anatomy of interest.The fluoro system may also be used to play back a run of fluoro in theimage stored during that playback. In some embodiments, this system mayalso be used to capture a sequence of frames over the respiration cycleor pulse cycle and play back a video as a stored image instead of astatic image.

Alternatively, in some embodiments, the virtual biplane may include oneor more renderings of the three-dimensional imaging of the anatomy suchas a segmented CT or MM image or intraoperative cone beam CT, IVUS, orultrasound. In some such embodiments, an image of the sensed medicalinstrument is placed within the three-dimensional rendering of theanatomy based on the registration of the medical instrument to theanatomical model. Multiple different registration methods may be used asdescribed above and in U.S. Pat. No. 9,014,851, the disclosure of whichis herein incorporated by reference in its entirety. Such embodimentsprovide multiple views for the user during navigation of the medicalinstrument without requiring live imaging. For example, the embodimentsof FIGS. 11A-B each shows a cone beam CT outline overlaid on a fluoroimage or alone on a black background. Similar imagery can additionallyor alternatively be created using imaging gathered preoperatively, suchas from a CT or MM. In these and other embodiments, overlaying arepresentation of the 3-D anatomy (be it an outline, filled solid area,3-D rendering, or composite of a stored fluoro image with 3-D imagery)on the background allows for the use of the 3-D data for guidancewithout additional fluoroscopy. This data can be interfaced with thenavigation system through either: a data connection between thenavigation system and the imaging system, or displaying frame grabbedvideo directly or composited with other imaging within the system. Inother embodiments, two display screens or two portions of the samedisplay screen show, from different perspectives, views of thethree-dimensional anatomical model with a simulation of the sensedmedical instrument positioned therein. In some embodiments, one of thedisplayed views is a simulated endoscopic view (also known as an “EndoView”), which provides the perspective of looking along the instrumentor from the front of the instrument within the three-dimensional modelof the anatomy. These or other views may supplement or form some of orall the virtual biplane.

Selection of Targets

As described elsewhere herein, an object of the present disclosure is toprovide systems and methods for robotic assisted driving of a medicalinstrument to a target within an anatomical space. In addition toacquiring images of the anatomical space, sensing and tracking themedical instrument, registering the coordinate system (i.e., referenceframe) of the medical instrument to the coordinate system of the images,and overlaying a representation of the medical instrument on the images,various methods of robotically assisted driving also requireidentification of the target. Together, the sensed information of themedical instrument and the identified location of a target can becombined to provide robotically-assisted driving modes. Various modesand embodiments of identifying the target are discussed in more detailbelow.

In the provided discussion, “targets” are referred to in a generic senseand may refer to a target position, a heading, an ostium shape, or otherelement or elements of interest to navigation. The target is generallythe 3-D center of an anatomical feature that the user would like toaccess, such as the ostium of a blood vessel coming off the aorta. Whilea blood vessel is frequently mentioned herein, a target could alsocorrespond to a feature of an implanted device such as an aorta aneurysmendograft or a fenestrated graft. In some embodiments, a target mayrefer to an anatomical feature such as an ablation site, the annulus ofa coronary valve, a suspect nodule or a tumor and may be within anylumen of the body including the airway, gastrointestinal tract, or otherlumen or within any organ accessed via a body lumen. When no additionalcontext is provided, “target” in this discussion refers to thethree-dimensional position at the center of the entrance of the featureto or through which the user wants to navigate.

In various embodiments, a target is designated in three-dimensionalspace via one or more user selections of the target position on multipleimaging views. In some embodiments, selection (i.e., designation) of thetargets involves a user interaction telling the system where the targetsare and how they are oriented. In other embodiments, the systemautomatically calculates where the targets would be based on knowninformation such as a three-dimensional CT or other target locations, ora combination of both. The three-dimensional imagery may be acquiredfrom a preoperative CT or via imaging during the procedure, such as viacone beam CT. The user may designate the target within an image using avariety of user input devices such as a mouse, trackball, buttons,joystick, or touchscreen. In some embodiments, the user manipulates auser input device to navigate a pointing device icon to the target inthe displayed images. The user may further manipulate a user inputdevice (for example, with a button push, mouse click, or finger tap) toselect the target. In one embodiment, a trackball and buttons are usedby the user to designate targets of interest. In other embodiments,other interfaces are utilized that also provide a means to identify andselect points in the images, such as, but not limited to, a computermouse, touchscreen, gesture device such as Xbox Kinect, joystick, etc.In some such embodiments, an interface allows the user to designate aposition on a three-dimensional CT or other three-dimensional image andthe system computes the target position, heading, ostium shape, vesselcenterline, or other information useful for navigation. This interfacemay be at an operator workstation or may be positioned bedside or in anyconvenient location and may be designed to operate in a sterile ornon-sterile environment based on the location chosen. In someembodiments, there may be multiple operator interfaces so that, forexample, an assistant can choose the target locations at a remoteworkstation and the physician can continue the navigation bedside.

In other embodiments, the imagery video from an imaging system, as shownfor example in FIG. 11B, may be frame grabbed by the system and thenprocessed using computer vision techniques to determine anatomicalfeatures of interest to create targets or provide additional guidance asthe user is creating the targets. For example, the system may detectbody lumen edges based on pixel density and automatically compute thediameters and centerlines of the lumens within the two views; such asystem may then allow the user to designate a single point in one imageto identify the target position, heading, and ostium size using theanatomical data reconstructed from the imagery.

In addition to the position of the target, it is useful to know thedirection or orientation of the lumen beyond the entrance or ostium(i.e., distal to the target). This direction, or “heading”, can help therobotically-assisted instrument driving system identify the bestapproach to enter the lumen. When a user needs to navigate through atarget, it is more complex than when a user needs to navigate to atarget. Navigation to a target often does not require a specific angleof approach whereas navigation through a target often requires themedical instrument to be lined up squarely with the target beforeadvancing through it. It is easier to enter a lumen if axially alignedwith the heading of the lumen rather than positioned perpendicular toit. Therefore, navigation through a target requires an understanding ofthe anatomy distal to the target and so requires more information,usually at least a second point. In some embodiments, the heading is asingle vector direction; in other embodiments, the heading may be adesignated or computed lumen centerline. In some embodiments, the userdesignates a position on the 3-D image and the imaging system computesthe shape and orientation of the target opening or the direction of thetarget lumen based on the 3-D geometry.

In other embodiments, a user is prompted to designate the shape of thelumen entrance so that this entrance can be better shown within the userinterface. In one embodiment, the target position, heading direction,and radius of the lumen are identified by the user within at least oneof the anatomical views and used to create a circle in space centered atthe target position and perpendicular to the heading direction. It isalso possible to use multiple points on the edge of the ostium or otherelement of interest to define a more complex shape such as an ellipse orany other closed shape. The user may designate these points on thedisplay screen. Alternatively, multiple points or line segments may besufficient to designate and define the shape of a vessel entrance. Oneembodiment of selecting the target is shown in FIGS. 10A-10C. In thedepicted example, a user is shown in FIG. 10A selecting a target in afirst image. The user may manipulate a user input device to navigate aset of crosshairs over the target. The user may also be able tomanipulate the user input device to change the size of the crosshairs soas to approximate the size of the target. As shown in FIG. 10B, once thetarget is selected in the first image, the system may determine a planealong which target is located, and a line projection may appear in thesecond image depicting the plane to facilitate target selection for theuser. The user is shown in FIG. 10B moving the crosshairs along the lineprojection and selecting the location of the target in the second image.As shown in FIG. 10C, following user identification of the target, thetarget (or a perimeter of the target) remains illuminated to facilitatevisualization of the target during instrument navigation. Instrumentnavigation is then depicted in FIG. 10D.

It is also possible to designate more than one target position (orheading or shape). As one example, different target positions aretypically required for the inner and outer members of a medicalinstrument. A first target may be designated by a user, and a secondtarget may be calculated based on the location, heading, or shape of thefirst target. For example, if the user identifies a target for the innermember, the system may automatically compute a separate target for theouter member. In some embodiments, the position of the separate targetfor the outer member is selected such that when the outer member isaligned with the outer member target, the inner member may beautomatically or semi-automatically positioned to align with the innermember target. In some embodiments, the heading of the inner membertarget relative to the heading of the outer member tip or other part ofthe virtual instrument may be used to determine the distance of theouter member target from the inner member target. In some embodiments,the system takes into consideration the known articulation length of theinner member to determine the outer member target position or heading.In some embodiments, anatomical information from 3-D imaging, such as apre-op CT, allows the system to better compute the outer member targetby taking into consideration lumen walls when computing the optimalouter member distance from the final target. In some embodiments, 2-Danatomical information from one or more images may be used to determinethe position of the outer member target by creating constraints inthree-dimensional space from the projection of the 2-D anatomicalinformation. In some embodiments, the user determines the outer membertarget based on the center line path from the 3-D dataset.

In some embodiments, the system is configured to perform a method tomodify the target after it is initially specified. For example, insteadof setting a new target, the user may be able to use a mouse or otherinput device to move an existing target in 2-D or 3-D space. In oneembodiment, the input device is used to move the target in a singleimage to fine tune its location in relation to the image or anatomyduring navigation. In other embodiments, the system automatically finetunes the target location based on other information such as 3-D imagingor other live intraoperative imaging such as IVUS.

Separate targets for both inner and outer members (e.g., the sheath andleader catheters) may be similarly reconstructed in this fashion takinginto consideration the anatomical shape. As an example, for a sharpvessel takeoff at an angle greater than 90°, such as the right renalvessel 1504 shown in FIGS. 15A-E, the system may determine the besttarget for the inner and outer members taking into consideration theangle of approach, the size of the lumen proximal to the target, thetakeoff direction of the sharp vessel, and the size of the lumen distalto the target.

In some embodiments, targets may be marked within the 3-D imaging systemmaking use of a registered pre-op CT, cone beam CT, or other imagingsystem. For example, the ostia of vessels may be marked in a 3-D volumeas described in U.S. Pat. No. 9,256,940, the disclosure of which isherein incorporated by reference in its entirety. These marks may beexported directly to a flexible instrument navigation system (e.g., theMagellan® system by Hansen Medical Inc.). In this manner, physicians mayuse their familiar registration, segmentation, and marking toolset, andthe data needed to improve navigation is exported to the navigationsystem as target or waypoint data.

In one example, a trackball or other user input device is used todesignate targets via the following sequence: (a) a pair of clicks, onein each of the two views of a virtual biplane, designate the targetposition for the inner member, (b) a pair of clicks, one in each of thetwo views of the virtual biplane, designate the target heading directionfor the inner member; (c) one click on the heading line designates anouter member target position; and (d) one click designates the radius ofthe ostium (or the size of any other target).

In some embodiments, a pair of clicks is required to designate a 3-Dposition of the target. After the first click on the first view (shownin FIG. 10A), that click (or 2-D position in the screen space) defines aline of possible positions in 3-D space based on the camera projection.As shown in FIG. 10B, that line is shown in the second view to aid inidentifying the corresponding position of that feature in the alternateC-arm angulation. Once the two clicks are processed, the lines based onthe camera projection are calculated and the closest point between thosetwo lines is used for the 3-D target position.

In other embodiments, when the heading direction of a target oranatomical feature needs to be defined as well as its 3D position, asecond point on the target (referred to as the “heading position”) isrequired and it may be designated in much the same way as the target 3-Dposition. Two clicks of the anatomical feature distal to the target fromtwo different views are processed to find the heading position on theanatomical feature, and the designated heading position is then usedwith the target position to calculate the heading direction. In someembodiments, a line corresponding to the possible heading is drawnbefore each point is clicked to help the user understand the possibleheading positions. In other embodiments, with a first click, the userinput device position is used to calculate the two-dimensional headingin the first image, and with a second click, the user input deviceposition in the second image is used to calculate the three-dimensionalheading based on the first two-dimensional heading in the first image.

In some embodiments, the size of the target (e.g., the ostium radius) iscalculated by using the closest distance between the line defined by the2-D clicked point and the target position. Many other interfaces may beused to set the target size or shape. For example, multiple clicksaround the edges of a target, as seen within 2-D space, may be used todefine a size and shape of the target in one view and to identify thetarget in the other view. In another embodiment, the target radius istraced, for example, using a mouse or trackball. In another embodiment,the size of the target radius is varied as the user input device ismoved up and down or left and right and is shown on the screen with adotted line until the user appropriately actuates the input device (forexample, with a click) to set the target size.

Once the targets are designated, multiple icons can be used to show thatthe target is designated. For example, in some embodiments, such asshown in FIGS. 10A-D, crosshairs or partial crosshairs may show wherethe target is, a line from the center of that target may show theheading of the lumen, a circle or other shape corresponding to theperimeter of a target may be highlighted, and/or a radar screen showingthe positioning of the target within the radar may be provided. When theuser switches the selection between different components of theinstrument, for example, between leader and sheath catheters, differenttargets may be displayed in different places corresponding to thedifferent members.

Robotic Assisted Driving

Once the current location of the medical instrument and the location ofthe target are known, the robotic driving system can help the usernavigate. Various embodiments of advanced driving modes are discussed inmore detail below. In some embodiments, the instrument driving system isconfigured to perform one of the disclosed advanced driving modes. Inother embodiments, the instrument driving system is configured toperform some of or all the disclosed advanced driving modes. In suchembodiments, the user may select the level of assistance or control theuser wishes to hand over to the robotic system. In various embodimentsprovided herein, the one or more advanced driving modes are encoded forin software saved to memory within the controller 34. The advanceddriving modes are referred to herein as “robotic assisted driving” or“robotic assisted navigation” modes. In various embodiments providedherein, user commands to the instrument driving system can be augmentedwith additional robot-determined movements to accomplish navigating themedical instrument to or through the target. In some embodiments, it ismost desirable to allow the robotic medical system to automaticallycommand the articulation direction and articulation magnitude of themedical instrument to arrive at or travel through the target. In otherembodiments, it may be desirable to allow the user to maintain at leastsome control over these motions while the robot assists. Therefore,there are various degrees of implementation of robotic assisted drivingwhich are presented herein.

In some embodiments, a computer-augmented driving mode is available tothe user. When such a mode is selected, the user may control instrumenttranslation (e.g., insertion and retraction), and while translation isoccurring, the robotic system may automatically control articulationmagnitude and direction (i.e., bend and roll) and provide additionalmovements of the instrument tip to help track the instrument to thetarget location. The automatic selection of the optimal articulationamount and roll direction by the controller prevents the user fromneeding to both perceive in three dimensions where the target is inrelation to the instrument and determine the amount of bend andarticulation needed to aim the instrument in that position in threedimensions. In a sense, the robotic instrument driving system (andspecifically, the controller or control algorithm of the robotic system)can assist the user in navigation even if the system does not know howto control all degrees of freedom of the instrument to achieve theuser's goal.

In another computer-augmented driving mode or embodiment, the usernavigates translation and one of articulation and roll, and the roboticsystem navigates the other of articulation and roll, as needed, toensure the user-commanded movements lead the medical instrument to theintended location. The exact amount of help or movement provided by thesystem may vary depending on the application.

One example of computer-augmented driving is depicted in FIGS. 12A-B.FIG. 12A provides a 2-D representation of an elongate instrument, as itwould appear to a user within a first view of a virtual biplane. FIG.12B provides a 2-D representation of the elongate instrument from adifferent point of view, for example, as it may appear within a secondview of the virtual biplane. In the depicted embodiment, the user's goalis to bend the virtual instrument 1200 (and the corresponding realinstrument) towards the target 1210 located at the 9 o'clock position inthe second view. As shown in FIG. 12B, in the user's first attempt tonavigate towards the target 1210, the user inadvertently commandedarticulation of the instrument towards 10 o'clock. In some suchembodiments of augmented driving, as the user commands roll or rotationof the instrument to find the target 1210, the system automaticallyupdates the mapping of the rotation input from the user to the desiredrotation and automatically rotates the instrument tip from the 10o'clock position to the 9 o'clock position. Alternatively, rather thanmapping a user input to the desired motion, the robotic system mayaugment the user inputs with additional motions to accomplish the tasks.For example, with the target 1210 designated at the 9 o'clock position,when the user commands articulation of the instrument, the roboticsystem automatically supplements the commanded articulation with a rollcommand to achieve the desired motion or navigate toward the target.

Robotic assisted driving has been explained above as helping oraugmenting user commands. Robotic assisted driving may also includeidentifying or automatically choosing movements such as the headingdirection of the instrument. The controller 34 may use the positionand/or heading of the instrument's articulation section and thelocation, shape, and/or heading of the target to automatically choosethe preferred roll plane and articulation magnitude, for example, inorder to reach the target or cannulate a target vessel most effectively.As discussed above, the exact implementation may vary based on theapplication and the user commands. For example, if the user iscommanding a roll motion as in FIG. 12B, then the automatic choosing ofthe preferred roll plane involves mapping the user input to a desiredmotion or adding additional robot-commanded movements to theuser-commanded movements to direct the instrument tip towards thetarget. If the user is commanding an insertion motion, then theautomatic choosing of the preferred roll plane includes augmenting theinsertion motion with rotational movement to direct the instrument tiptowards the target. Likewise, the automatic choosing of a preferredarticulation magnitude may consist of mapping an articulation user-inputcommand to a robot-determined desired articulation movement to therebydirect the instrument tip towards the target when the user commandsarticulation. Additionally or alternatively, robotic assisted drivingmay include supplementing an insertion motion with appropriaterobot-determined articulation movements to direct the instrument tiptowards the target.

Additionally or alternatively, in some embodiments of robot assisteddriving, the controller 34 commands the system to display a recommendedpath or shape of the instrument to the user. One example is provided inFIG. 13. As depicted, the recommended path 1310 through the anatomy 1300may be denoted with a hidden or dashed line or other suitable marking.The recommended path 1310 depicts an optimal or suitable path within theanatomical image for getting the virtual instrument 1320 from itscurrent position to the target. Such a display allows the user to seehow the instrument should be oriented. In some embodiments, the user canthen control the instrument, using the path 1310 as a guide. In somesuch embodiments, the controller 34 is configured to suspend movement ofthe instrument and notify the user automatically if the instrumentdeviates from the recommended path. In another embodiment, the systemmay automatically reduce the insertion speed if the instrument hasdeviated from the path 1310 or if articulation has not yet reached adesired amount in order to minimize force on the anatomy. In alternativeembodiments, the controller 34 may ensure that the instrument followsthe recommended path by supplementing or adjusting the user-commandedmovements with robot-determined course-correcting movements. In otherembodiments, the controller 34 computes one or more insertion,articulation, and/or rotation movements needed to keep the instrumenttip on the path 1310. In some embodiments, the controller 34 commandsthe instrument driver 22 to execute motor actuations needed to implementthe one or more movements. In some such embodiments, the controller 34and the instrument driver 22 work together to drive the instrument alongthe recommended path 1310 while an auto-pilot feature of the user inputdevice is actuated. In some such embodiments, the user must activate aninput device in order to continue progress. At any time, the user maydisengage the input device to stop all motion.

The recommended path 1310 may be derived from anatomical informationprovided by the imaging subsystem. The anatomical information may takethe form of a 3-D model of the anatomy, and the recommended path mayequal the centerline of a segmented body lumen. In other embodiments,the anatomical information may be in a two-dimensional form such asframe grabbed images from the imaging subsystem. In some embodiments,the anatomical information is used to adjust the computed movements ofthe instrument as it navigates the anatomy by choosing articulation androll values that keep the instrument away from the lumen walls. In someembodiments, the anatomical information allows the controller 34 tobetter determine when to insert one or more members of the instrument toachieve the best shape, maintain a sufficiently large distance away fromthe lumen wall, and/or enable the instrument to move in the lumen withminimal resistance.

It can be important to provide feedback to the user to let the user knowthat the instrument is progressing correctly. In some embodiments,visual indicators are provided to help the user understand therelationship between the instrument and the targets and improve controlover the instrument. Visual indicators may be provided to show that thetarget algorithm is converging on a solution and aiming the instrumenttowards the target position. In some embodiments, this can be indicatedusing color on or around the target. For example, in one embodiment, ared border around the target is displayed when the instrument is farfrom the target, yellow is displayed when nearing the target, and greenis displayed when the instrument is aligned with the target.Additionally or alternatively, in some embodiments, convergence on asolution that aims the instrument correctly on the target is depictedwith a circle or other shape centered at the target position with aradius equal to the distance of the heading of the instrument from thetarget position. In other embodiments, such as depicted in FIG. 9, ageometric shape such as a square or circle changes in size to indicatehow well the desired path is achieved.

In some embodiments, desirable paths for both the outer member 924 andthe inner member 922 of an instrument 920 may be indicated on the screenwith lines, dots, geometric shapes, or other imagery. For example, inFIG. 9, the following are depicted: a rectangle 940 around the target915, a rectangle 942 adjacent the location of the articulation sectionof the inner member 922, and a rectangle 944 adjacent the location ofthe articulation section of the outer member 924. In one embodiment, therectangle 940 around the target 915 changes in color or size to indicatethe extent to which the instrument is following a suitable path to thetarget. In one embodiment, a dot representing a heading direction ofeach articulation section appears within the rectangles 942, 944 whenthe articulation sections are in a suitable position in the anatomicalspace. In some embodiments, such as FIG. 10D, the dot indicates theprojection of a distal tip of the instrument onto the plane of theostium circle or other target plane (similar to a laser targeting systemon a rifle placing a dot wherever the rifle is aimed). The dot may beprovided to show the relationship between the tip of an articulationsection and the target.

In some embodiments, shown for example in FIG. 10D, a “target radar” isprovided as a visual indicator to facilitate understanding of theinstrument position relative to the target. In the image, the crosshairsand circles show the space of articulation for the instrument (includingarticulation magnitude and articulation direction or “roll”). This issimilar to polar coordinates. The “x” within the “radar” corresponds tothe location of the instrument. Its position relative to the centercorresponds to the heading direction and the circle corresponds to theoutline of the vessel ostium or other target plane. In such embodiments,when the instrument is rolled, the “x” moves around the circles; whenthe instrument is articulated more, the x moves toward the outside ofthe circle, and when the instrument is relaxed, the x moves toward thecenter. Using this indicator, a user can determine which way toarticulate and roll the instrument to align it with the target. A usermay also use the targeting radar to discern which direction theinstrument is aiming within the target ostium. In a sense, this works asa simplified endoscopic view of the instrument direction and target.Alternatively, a 3-D version of the indicator may be used that draws theindicator in 3-D space; the version may additionally draw the ostiumwith a projected heading of the instrument on the ostium surface definedby the ostium circle. A user may also use the targeting radar to discernwhich direction the instrument is aiming within the target ostium. In asense, this works as a simplified endoscopic view of the instrumentdirection and target. Alternatively, a 3-D version of the indicator maybe used that draws the indicator in 3-D space; the version mayadditionally draw the ostium with a projected heading of the instrumenton the ostium surface defined by the ostium circle.

As also shown, for example, in FIG. 10D, some embodiments may display ashadow instrument representing the position the instrument is expectedto assume if a user-entered movement command or the next proposedrobot-determined command is implemented. Such an embodiment effectivelyprovides a verification feature and requires another click or other userinput before the system proceeds with implementing the commandedmovement.

In some embodiments, shown in FIG. 13, dotted lines or other indicatorsare provided to show the ideal instrument path or the boundary ofsuitable paths for the instrument, and the user is able to see thedifference between the actual instrument shape and the ideal instrumentpath to identify if deviation is occurring. In some embodiments, oncethe target is set, the system may allow the user to navigate withlimited robotic assistance but will display one or more visualindicators indicating whether the instrument is following a suitablepath. Visual cues such as colors, geometric shapes, diagrams, and/or aseries of lights (for example, similar to airplane landing lights) maybe displayed to show the relationship between the instrument and thetarget or the recommended path and the current path.

In some embodiments, visual feedback is additionally provided to showwhen the various advanced driving modes have been enabled for variouscomponents of the instrument. For example, when the system is in aninner member driving mode, the target for the inner member may be theonly visible target or may be specially highlighted, and similarly, whenthe system is in an outer member driving mode, the target for the outermember may be the only visible target or may be specially highlighted.In some embodiments, icons to the side of the virtual instrument orlighting on the user input device may indicate when various assisteddriving buttons are enabled.

In another embodiment, the system may display an appropriatearticulation magnitude and/or direction to the user for the user tofollow, in effect, providing textual or graphical turn by turndirections or step-by-step instructions, for example, telling the userwhich user inputs to select and when. While in such a “Driving Wizard,”which may include a sequence of messages (or dialogues or text boxes orsymbols) guiding the user towards the target, undesirable motions mayalso be blocked to ensure that the user drives the instrument correctlyor consistently with the guidance. In a similar embodiment, once thetarget is identified, the system may allow the user to drive, but thecontroller may create an alert and/or automatically stop movement of themedical instrument if a user command would move the instrument in such away that reaching the target would become difficult or impossible.

In still another embodiment or driving mode, referred to asrobot-controlled navigation, the system fully controls navigation of themedical instrument including one or more of the articulation, roll, andtranslation. This is considered automated navigation and is madepossible once a target position, heading, and lumen size are set. Insuch a mode, the system may control all movement of the medicalinstrument once the target is identified; in some embodiments, thesystem may control all movement while the user is selecting anassociated user input command. For example, in some embodiments, thecontroller may calculate the amount of instrument articulation,rotation, and translation needed to reach the target in an optimal way.The translation, articulation, and rotation may be optimized so that thetip of the instrument is aligned with the target position as well as thetarget heading. The translation, articulation, and roll may also beoptimized so that the shape of the medical instrument does not collidewith the anatomy, if the system is able to make use of three dimensionalpreoperative imaging, other three-dimensional imaging, ortwo-dimensional imaging showing the outline or projection of the 3-Danatomy. In some embodiments, motion stops when the user stops actuatingan automated driving button or other user input device. In otherembodiments, it may be possible for the system to drive the instrumentautomatically even though the user has released the user input device.Some embodiments may automatically alternate between translating theinstrument and modifying the articulation of the instrument. Otherembodiments may automatically modify the articulation of the instrumentas it is translated or allow the combination of user-commandedtranslation motions with robot-commanded control over articulation androll. Other embodiments of robotic controlled navigation allow the userto specify the shape of the anatomy in the region so that the system canbetter calculate the translation, articulation, and/or roll of theinstrument to align with the target. Some embodiments may make use ofmany of these sensing and navigation modalities to automatically computeall articulation and translation of the instrument to achieve a systemthat is able to navigate the instrument along a prescribed trajectory ora centerline of a body lumen. Methods for extracting the centerline froman image volume are described, for example, in U.S. Pat. No. 9,129,417,the disclosure of which is herein incorporated by reference in itsentirety. This centerline generated from the 3-D volume may be used asthe target for the automated robotic driving algorithm. In someembodiments, the navigation system may analyze the 3-D data set importedfrom a pre-op CT, MRI, or cone beam CT directly to compute the sequenceof targets in three dimensions. In other embodiments, the navigationsystem processes one or a small number of 2-D images from an imagingsystem to improve the targeting algorithms. For example, each singleimage provides a two-dimensional constraint on the lumen shape when theoutline in the 2-D image is projected into three dimensions; suchinformation can be used to inform the target shape, heading, orlocation. If many images are acquired, such as during a rotation of theC-arm, the navigation system can reconstruct the 3-D shape using 2-D to3-D reconstruction techniques (similar to how a CT is reconstructed).

Robotic assisted driving techniques may be used to access any anatomicaltarget. One non-limiting example includes the crossing of an occlusionin a blood vessel. In an occluded blood vessel, there is no bloodflowing so it is not possible to image the anatomy using an angiogramunder fluoroscopy. However, recent developments in CT scanning canidentify the thrombus or calcium making up the occlusion and canidentify the centerline of the occluded vessel. This three-dimensionalinformation of the centerline of vessels can then be used to generate asequence of targets that comprise the catheter trajectory. The roboticassisted driving algorithm of some embodiments is configured to usethese targets as a path and navigate from the beginning of the occlusionto the location where the vessel reconstitutes by following thiscenterline while crossing the occlusion. In some embodiments, therobotic control system may automatically extract the centerline data andfollow it.

In alternative embodiments, robot-controlled navigation can occurintermittently; for example, a user may begin driving the medicalinstrument and select the robot-controlled navigation on occasion inorder to have the system make path corrections. At times, it isimportant to prevent the computed articulation or roll from articulatingthe medical instrument in a constrained situation, because it isimportant to prevent the instrument from pressing into the anatomy.Similarly, it is important to prevent a computed insertion frominserting the instrument into the anatomy. Some embodiments may includea subsystem that monitors the instrument motion in relation toinstrument commands such as articulation and insertion. By modeling thecommands and comparing them to the measured catheter shape, the systemis able to determine whether it is likely that the medical instrument iscontacting the anatomy. Some embodiments may also calculate, based onthis difference between the commanded shape and the measured shape, anestimate of the force applied on the instrument by the anatomy (andlikewise, the force of the instrument on the anatomy). If the computedforce gets large, the system may prevent further motion or cause arelaxation of the instrument to reduce this force on the anatomy. Someembodiments may also provide a message to the user or prevent assistednavigation when the computed force becomes too large. Some embodimentsmay also compute this force even when the user is not using assistednavigation to prevent the user from inadvertently causing too much forceon the anatomy during navigation.

During navigation, the instrument commands are represented as anarticulation magnitude, roll angle, and insertion length, and thereforethe controller directly modifies the commands to facilitate the driving.The commands are modified to serve different tasks at different stagesof driving. For example, in the beginning, the focus may be oncannulating a vessel, and the controller may focus on aiming at thetarget. As the procedure progresses, and the flexible instrumentapproaches the target, the controller may focus on bringing theinstruments through the target, requiring a different strategy thanaiming the instruments at the vessel. In both cases, the modifyingcommands must be defined in the same coordinate system as the commandsissued by the physician. In various embodiments, the desiredarticulation and roll commands are defined in a frame of reference ofthe instrument sensors, often located at the base of the articulationsection. In some embodiments, the frame of reference may be computedfrom the virtual instrument shape, which can be defined by a combinationof one or more of the sensor data, the articulation command to themedical instrument, a simulation of the instrument dynamics, and a modelof instrument behavior. Once the target is also identified in this frameof reference, the desired articulation and roll commands are generatedto aim the instrument at the target. In some embodiments, a searchalgorithm is employed to find the optimal articulation and roll angle.In another embodiment, an optimization procedure can determine the bestarticulation, roll, and insertion.

In another embodiment, the frame of reference of the instrument is nolonger attached to the base of the instrument's articulation section,but is instead calculated based on the shape of the virtual instrumentconstructed from sensor measurements as well as commands to theinstrument. The coordinate frame at the distal tip of the instrument isdirectly measured by the sensors, but the frame at the base of thearticulation section is calculated from inverting the kinematics thatdescribes the relationship between the articulation magnitude and theposition of the distal tip. The resulting frame of reference is nolonger placed at the base of the articulation section, but instead takesinto consideration the shape of the instrument and the command thatcaused the instrument to take the shape. This may lead to fasterconvergence and improved targeting performance.

In open loop control, calculations from the sensors are used with thetarget location data to compute a single direction to move the medicalinstrument. This approach has the advantage of control stability, butvariations in instrument behavior may prevent the instrument from aimingdirectly at the target. In closed loop control, the system takes intoconsideration the sensed position of the instrument as it moves andadjusts the instrument command accordingly to make the aim of theinstrument converge on the target. In one embodiment, a polar coordinatesystem such as the targeting radar indicator displayed in FIG. 10D isused to compute the difference between the center of the target and thecurrent heading of the instrument. A change in articulation and rollangles can be identified and added to the current instrument command tobetter align the instrument with a target. In various embodiments, largecommanded changes may need to be divided into a series of smaller stepsto prevent overshooting the target. On the other hand, a minisculechange or step may be magnified to overcome non-linear characteristicsof the instrument, such as friction, dead-zone, or slack in a pullwire,to ensure that the instrument exhibits noticeable motion during assisteddriving. In addition to the setting time, a distance threshold may beimplemented to stop further modifications to the command if the aim ofthe instrument is close enough to the target. The threshold prevents theinstrument from unnecessarily overshooting the target and keeps the aimof the instrument from drifting away from the target once the target hasbeen reached. In various embodiments, once a desired instrumentarticulation and roll are computed, the controller 34 breaks them intodriving commands. In one embodiment, the instrument may be commanded tofirst relax if it needs to change roll direction more than 90 degreesinstead of rotating the instrument. Once the instrument is fullyrelaxed, the roll angle can be set directly so that the instrument bendsin the desired roll direction. This is similar to the adaptive cathetercontrol strategies outlined in US Publ. No. 2016/0213884, the disclosureof which is herein incorporated by reference in its entirety.

Robotic assisted driving may involve the task of advancing an instrumentformed of a plurality of members towards or through a target. In someembodiments, one or more of the above systems and methods are used torobotically assist driving an inner member through the target. Once theleading, inner member is passed through the target, an enhanced roboticassisted driving algorithm and method may be implemented so that anycoaxial outer members “follow” over the leading member. Duringnavigation of the outer member, the shape of the inner member providesan ideal path for the outer member to follow.

In some embodiments, “following” involves inserting the outer membertowards the target while articulating the outer member in the directionof the target, thereby following a path similar to the ideal path. Inother embodiments, additional steps are needed to ensure accuratefollowing and to avoid prolapse or loss of wire or instrument position.There is a risk of prolapse any time a flexible instrument changesdirections during insertion. Prolapse is a situation where insertion ofthe instrument causes a proximal portion of the instrument to continuemoving in the direction of a previous insertion command and bulge awayfrom a target instead of changing directions and advancing with thedistal tip toward the target. In FIG. 13, for example, a prolapse in theleader catheter 1324 is shown as a bulge above the level of the targetostium 1302; in this situation, further insertion of the leader catheter1324 would cause it to buckle further up into the aorta 1304 andultimately pull the tip of the leader catheter 1324 and the guidewire1322 out of the ostium. This situation is important to avoid because itcan add significant delays to a procedure. The prolapse of the leadercatheter 1324 or other inner member is often controlled by theorientation of the sheath catheter 1326 or other outer member. Forexample, in FIG. 13, the likely cause of the prolapse in the leadercatheter 1324 is the fact that the tip of the sheath 1326 is aimedsignificantly away from the ostium 1302 of the vessel and the change indirection was too great. If the sheath catheter 1326 were insteadarticulated to point more towards the ostium 1302, the leader catheter1324 would be directed more towards the target and would have moresupport as it is inserted. The additional support enables the insertionmotion to move the leader catheter 1324 through the ostium 1302 and intothe vessel instead of further up into the aorta 1304.

Additionally or alternatively, in some embodiments, avoiding prolapseinvolves retracting an inner member 1324 before articulating the outermember 1326 towards the bend or target. Because the controller 34 isaware of the shape of the instrument 1320, some embodiments make use ofthis information to automatically avoid these prolapse situations or, ifthey are detected, to move the instrument 1320 in such a way that theprolapse is removed. In some embodiments, the controller 34 isprogrammed and the system configured to detect prolapse within theinstrument and notify the user and/or stop motion.

In some embodiments, avoiding prolapse involves relaxing the outermember 1326 articulation then articulating the outer member 1326 in thepath direction of the guidewire 1322 or inner member 1324. The path ofthe guidewire 1322 or inner member 1324 may be defined by the virtualinstrument shape generated from the sensor data. The shape of thevirtual instrument constructed from the sensor measurements is naturallysmooth and closely mimics that of the real instrument. The virtualinstrument provides sufficient information to generate proper commandsfor the real instruments.

In some embodiments, instrument commands may be further refined based onthe position and heading of the target as well as the shape of aninstrument portion. In one embodiment, the controller 34 determines anappropriate articulation angle θ_(cmd) for the outer member based, inpart, on the shape of the inner member. As shown in FIG. 14, todetermine the appropriate articulation angle θ_(cmd), the average innermember shape in a local area is calculated, and the controller solvesfor the articulation angle θ_(cmd) that will make the outer membercurvature match the average inner member shape the best. The controllermay then command the instrument driver to achieve this articulationangle. For example, in FIG. 14, a first heading value H₁ is sampled at afirst location L₁ proximal to the tip of the outer member 1404, and asecond heading value H₂ is sampled at a second location L₂ distal to thetip of the outer member 1404. The heading values are averaged to findthe average shape of the inner member 1402 in the sampled region,denoted as θ_(track). The length of the articulation section is denotedas d, and the length of the sampled portion (i.e., the length of theinner member 1402 between L₁ and L₂) is denoted as d′. The appropriatearticulation angle θ_(cmd) is then calculated using the followingproportion equation: θ_(cmd)=θ_(track) (d/d′). In another embodiment,the curvature of the articulation section is determined from averagingthe curvature across portions of the virtual instrument.

In various embodiments, the algorithms used to calculate thearticulation and roll of an instrument in assisted driving may need totake into consideration the pulsatile flow in the arteries as well asheart and breath motions. In some embodiments, biological motions of thepatient may be predicted and used to improve the performance of assisteddriving. For example, in the images, the target may appear to move insync with the patient's motion. Motion of the target may be sensed basedon the instrument motion, live imaging such as fluoroscopy, or userinput. The systems and methods of some embodiments detect and compensatefor this cyclic motion, stabilizing the algorithm to converge faster.Some embodiments use an adjusted or moving target during computations ofthe translation, articulation, and/or roll.

One embodiment of robotic-assisted driving is provided in FIGS. 15A-15E.In the provided illustrations, a virtual instrument 1510 is superimposedon the anatomical image 1500. The virtual instrument 1510 includes avirtual guidewire 1512, a virtual inner member 1514, and a virtual outermember 1516. In the provided example, the anatomy is representative of apatient's left renal artery branching from the aorta when viewed from ananterior/posterior projection, but similar branches are present in otherlumens within the body. The dashed line 1520 is provided in the visualdisplay of some embodiments to show the desired target path of theinstrument 1510 from its current position to the user-set target 1530.The dashed line 1520 (i.e., the desired path) may be positioned at thecenterline of the body lumen when within a relatively large or straightlumen 1502 and may follow a lowest energy curve or optimal path based onthe bending radius of the instrument as it navigates lumen branches1504.

The end target 1530 for the instrument 1510 is in the side branch vessel1504. If this target endpoint were established as the first target forboth the inner member 1514 and the outer member 1516, then both memberswould bend towards the target as shown by the solid arrows of FIG. 15Aand deviate from the desired path. This would lead to both memberscolliding with the wall of the aorta 1502. To prevent such an outcome,in various embodiments of robotic-assisted driving, the controller 34 isconfigured to determine and set one or more intermediate targets thatthe instrument components aim for along their path to the finaluser-designated target. For example, as shown in FIG. 15B, in thedepicted progression, the controller 34 has set a first target point1532 at the ostium of the vessel. This is a suitable target point forthe inner member 1514 in this configuration but not for the outer member1516. If the outer member 1516 were bent towards the target 1532, thenthe inner member 1514 would need to make a very sharp bend as shown bythe dashed line 1522. Instead, it is preferred to set up an additionaltarget point for the outer member, as shown by the visual indicator 1534in FIG. 15C. The target point 1532 for the inner member remainsunchanged. Both the target points 1534 and 1532 are locatedapproximately on the preferred trajectory or desired target path 1520 ofthe instrument. Once the target 1534 for the outer member is set, anymovements of the outer member 1516 determined and commanded by the robotwill be selected to direct the tip of the outer member 1516 towards thetarget as shown by the long arrow. Similarly, once the target 1532 isset for the inner member 1514, any movements of the inner member 1514determined and commanded by the robot will be selected to direct theinner member 1514 towards that target 1532 as shown by the short arrow.

It is worth noting that FIGS. 15A-E depict a single 2D image of thetarget and the instrument for simplicity. In various embodiments of thesurgical environment, a second image of a different projection is alsoprovided. The concepts described here for a single 2D image also holdtrue in 3D. The inner and outer members are both assigned a target in 3Dspace and are commanded to move towards those targets.

As shown in FIG. 15C, once the inner member 1514 and the outer member1516 are aligned towards their targets and are close to the targets, theguidewire 1512 may be advanced through the inner member target 1532.Once the guidewire 1512 is advanced through the target 1532, furtherforward motion of the instrument may be programmed to follow theguidewire. The inner member 1514 may need to be relaxed or straightenedas it advances beyond the apex of the bend to follow the guidewire 1512;however, in some embodiments, it may not be necessary to set new targetsfor the instrument.

In other embodiments, if further navigation into complex anatomy isrequired, the inner member target may be relocated, as shown in FIG.15D. In the illustration, the new target 1536 is set at the distal endof the branch lumen while the outer member target 1534 remains outsidethe ostium of the vessel so that the outer member 1516 provides adequatesupport on the desired dashed line. The outer member 1516 eventually iscommanded by the controller 34 to “follow” the inner member 1514, asshown in FIG. 15E. As with the inner member 1514, once the outer member1516 advances beyond its target 1534, the controller 34 may direct theouter member 1516 to follow the inner member 1514 or the guidewire 1512.Alternatively, a new target point may be set for the outer member 1516.

In other embodiments, such as ablation procedures, the control algorithmof the controller 34 may be set up such that the instrument 1510 neveradvances passed the target. In ablation procedures, the goal is to get acatheter tip to a target and it might be desirable as a safety measureto never allow the catheter or other instrument to extend beyond thetarget point. This would reduce the risk of inadvertent vesselperforation.

As discussed above, the degree of operator involvement in the setting ofthe targets and the driving towards or through the targets may vary. Inone preferred embodiment, the operator identifies the end target, therobotic system identifies a centerline or lowest energy path, theoperator control insertion of the guidewire and insertion and bend ofthe inner member, and the controller 34 automatically determines andcontrols the roll of the inner member and all movement (i.e., insertion,bend, and roll) of the outer member.

While multiple embodiments and variations of the many aspects of theinvention have been disclosed and described herein, such disclosure isprovided for purposes of illustration only. Many combinations andpermutations of the disclosed systems and methods are useful inminimally invasive medical intervention and diagnosis, and the systemsand methods are configured to be flexible. The foregoing illustrated anddescribed embodiments are susceptible to various modifications andalternative forms, and it should be understood that the inventiongenerally, as well as the specific embodiments described herein, are notlimited to the particular forms or methods disclosed, but also cover allmodifications, equivalents, and alternatives falling within the scope ofthe appended claims.

What is claimed is:
 1. A robotic medical system comprising: Arobotically controlled medical instrument; an instrument driver coupledto the medical instrument; a console including a user input device,wherein the console is configured to drive the medical instrumentthrough an anatomical space based on user commanded data received; adisplay for displaying a virtual representation of the medicalinstrument and a path depicting a recommended non-linear shape andorientation for navigation of the medical instrument from a currentposition towards a target location for the medical instrument on animage of the anatomical space; and a controller configured to: detect alocalization sensor of the medical instrument and a location, anorientation, and a shape of the medical instrument based on thecommanded data, register the medical instrument to the anatomical space,signal the display screen to display the virtual representation of themedical instrument on the image of the anatomical space based on thelocation, the orientation, and the shape of the medical instrument andthe commanded data received as the medical instrument is driven throughthe anatomical space, receive an identification of a target location forthe medical instrument, and signal the display screen to display, withthe virtual representation of the medical instrument on the image of theanatomical space, the path in the form of a line depicting therecommended non-linear shape and orientation directed from the currentposition towards the target location for the medical instrument.
 2. Themedical system of claim 1, wherein the medical instrument comprises aflexible instrument.
 3. The medical system of claim 2, wherein themedical instrument receives an imaging tool therein.
 4. The medicalsystem of claim 2, wherein the medical instrument receives a diagnosticand/or therapeutic tool therein.
 5. The medical system of claim 3,wherein the medical instrument includes one or more pullwires disposedtherein for articulating the medical instrument.
 6. The medical systemof claim 5, wherein the instrument driver comprises one or more pulleystherein about which the one or more pullwires can be wound.
 7. Themedical system of claim 6, wherein the localization sensor comprises afiber optic sensor therein.
 8. The medical system of claim 7, furthercomprising an imaging subsystem.
 9. The medical system of claim 8,wherein the imaging subsystem comprises a CT or fluoroscopy system. 10.The medical system of claim 1, wherein the localization sensor comprisesan EM sensor.
 11. The medical system of claim 10, wherein the medicalinstrument includes a pair of EM sensors.
 12. The medical system ofclaim 11, wherein each of the EM sensors comprises at least a 5-DOFsensor.
 13. The medical system of claim 1, wherein the display isconfigured to display an anatomical target.
 14. The medical system ofclaim 13, wherein the display is configured to display more than oneimage of the anatomical space.
 15. The medical system of claim 13,wherein the display is configured to display a circle or ring indicatorwith the representation of the medical instrument.
 16. A robotic medicalsystem comprising: a robotically controlled medical instrument, whereinthe robotically controlled medical instrument comprises a flexibleinstrument for receiving a diagnostic tool; an instrument driver coupledto the medical instrument; a console including a user input device,wherein the console is configured to drive the medical instrumentthrough an anatomical space based on user commanded data received; adisplay for displaying a virtual representation of the medicalinstrument and a path depicting a recommended non-linear shape andorientation for navigation of the medical instrument from a currentposition towards a target location for the medical instrument on animage of the anatomical space; and a controller configured to: detect alocalization sensor of the medical instrument and a location, anorientation, and a shape of the medical instrument based on thecommanded data, register the medical instrument to the anatomical space,signal the display screen to display the virtual representation of themedical instrument on the image of the anatomical space based on thelocation, the orientation, and the shape of the medical instrument andthe commanded data received as the medical instrument is driven throughthe anatomical space, receive an identification of a target location forthe medical instrument, signal the display screen to display, with thevirtual representation of the medical instrument on the image of theanatomical space, the path in the form of a line depicting therecommended non-linear shape and orientation directed from the currentposition towards the target location for the medical instrument.
 17. Themedical system of claim 16, wherein the instrument driver comprises asplayer with one or more pulleys received therein.
 18. The medicalsystem of claim 17, further comprising an imaging subsystem including aCT system.
 19. The medical system of claim 16, wherein the display isconfigured to display an anatomical image obtained with the assistanceof an injectable contrast agent.
 20. The medical system of claim 16,wherein the instrument comprises a fiber optic sensor.